Note

Adds S3-backed storage for dataset files/images with presigned upload/download, JWT proxy, and end-to-end UI/backend changes while keeping GridFS compatibility.

• Storage/S3 Integration:
  • Introduces S3DatasetSource (upload by buffer, get/put/stat, delete by key/prefix, list) and TTL utils; supports dataset/chat/avatar buckets.
  • Adds JWT-signed proxy endpoint GET /api/system/file/[jwt] to stream S3 objects.
  • Handles mixed keys: recognizes S3 dataset/... vs GridFS IDs across delete/read paths.
• APIs:
  • New endpoints: POST /core/dataset/presignDatasetFilePostUrl, POST /core/dataset/presignDatasetFileGetUrl (zod schemas in global/core/dataset/v2/api.ts).
  • Updates dataset create/import/image upload flows to S3 (presigned POST), including parsing-uploaded images and removing TTLs after use.
  • Read/parse pipeline now uploads parsed images to S3 and replaces refs; link/file readers adapted to S3 buffers.
• Dataset Processing:
  • Collection parsing/vector queues adapted for S3 files; manage TTLs and parsed-image cleanup by prefix; support image collections via S3.
  • Read/source sync and preview chunk generation accept datasetId and S3 keys.
• Permissions/Auth:
  • File auth supports S3 dataset keys; dataset citations/images use signed URLs; adds JWT sign/verify helpers.
• Schema/Types:
  • fileId in collections is now string (S3 key or GridFS id); collection file metadata simplified to { filename?, contentLength? }.
  • Adds zod schemas/types for presign params and S3 helpers.
• Frontend:
  • Markdown image component resolves dataset/... and chat/... via presigned URLs.
  • Dataset import (local) uploads via presigned POST; metadata card shows decoded filename and S3 content length; open-source now uses presigned get.
• Misc:
  • Removes Mongo-only read path; refactors S3 base bucket (copy/delete, put/get/stat, presign policy metadata); i18n strings updated.

^{Written by Cursor Bugbot for commit 31bfab7. This will update automatically on new commits. Configure here.}